Sensor fusion by diffusion maps

Data fusion and the analysis of high-dimensional multisensor data, are fundamental tasks in many research diciplines. In this work we propose a unified embedding scheme for multi sensory data, which is based on the recently introduced diffusion framework. Our scheme is purely data-driven and assumes no a-priory knowledge of the underlying statistical or deterministic models of the different data sources. Our approach is based on embedding separately each of the input channels and combining the resulting diffusion coordinates. In particular, as different sensors samples similar phenomena with different sampling densities, we apply the density invariant Laplace-Beltrami embedding. This is a fundamental issue in multisensor acquisition and processing, overlooked in prior approaches. In order to verify the efficiency of our approach, we apply it to multisensory statistical learning and clustering applications, such as spoken-digit recognition and multi-cue image segmentation. For both applications we experimentally show that using the unified multisensor embedding, allow better performance than the one achieved by any single sensor.